<!DOCTYPE html>
<html>

<head>
	<meta charset="utf-8">
	<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
	<meta name="theme-color" content="#33474d">
	<title>shell 练习-日志汇总 | 失落的乐章</title>
	<link rel="stylesheet" href="/css/style.css" />
	
      <link rel="alternate" href="/atom.xml" title="失落的乐章" type="application/atom+xml">
    
</head>

<body>

	<header class="header">
		<nav class="header__nav">
			
				<a href="/archives" class="header__link">Archive</a>
			
				<a href="/tags" class="header__link">Tags</a>
			
				<a href="/atom.xml" class="header__link">RSS</a>
			
		</nav>
		<h1 class="header__title"><a href="/">失落的乐章</a></h1>
		<h2 class="header__subtitle">技术面前，永远都是学生。</h2>
	</header>

	<main>
		<article>
	
		<h1>shell 练习-日志汇总</h1>
	
	<div class="article__infos">
		<span class="article__date">2017-10-12</span><br />
		
		
			<span class="article__tags">
			  	<a class="article__tag-link" href="/tags/shell练习/">shell练习</a>
			</span>
		
	</div>

	

	
		<p>我有一个日志（php的slow_log) 几乎每分钟都有输出信息。需要我们来写个脚本分析一下它，目的就是为了归类汇总，按照它们出现的频次做个排序，假如日志是每天0点5分清空，那么按照每小时一次汇总分析该日志，最后在第二天0点0分时，再汇总一下整天的日志，怎么写呢？ 下面我给出一个日志样例和我写的分析汇总脚本供参考。日志样例：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div></pre></td><td class="code"><pre><div class="line">[24-Oct-2013 00:05:39]  [pool www.lishiming.net] pid 19101</div><div class="line">script_filename = /data/release/www.lishiming.net/forum.php</div><div class="line">[0x00007f9279237e98] mysql_unbuffered_query() /data/release/www.lishiming.net/<span class="built_in">source</span>/class/db/db_driver_mysql.php:147</div><div class="line">[0x00007f92792377a8] query() /data/release/www.lishiming.net/<span class="built_in">source</span>/class/discuz/discuz_database.php:136</div><div class="line">[0x00007f9279236f40] query() /data/release/www.lishiming.net/<span class="built_in">source</span>/class/table/table_forum_thread.php:932</div><div class="line">[0x00007f92792365e8] increase() /data/release/www.lishiming.net/<span class="built_in">source</span>/module/forum/forum_viewthread.php:1034</div><div class="line">[0x00007f9279218f48] viewthread_updateviews() /data/release/www.lishiming.net/<span class="built_in">source</span>/module/forum/forum_viewthread.php:353</div><div class="line">[0x00007f9279218050] +++ dump failed</div><div class="line"></div><div class="line">[24-Oct-2013 00:05:39]  [pool www.lishiming.net] pid 19754</div><div class="line">script_filename = /data/release/www.lishiming.net/forum.php</div><div class="line">[0x00007f9279237938] mysql_query() /data/release/www.lishiming.net/<span class="built_in">source</span>/class/db/db_driver_mysql.php:147</div><div class="line">[0x00007f9279237248] query() /data/release/www.lishiming.net/<span class="built_in">source</span>/class/discuz/discuz_database.php:136</div><div class="line">[0x00007f9279236dc0] query() /data/release/www.lishiming.net/<span class="built_in">source</span>/class/discuz/discuz_database.php:100</div><div class="line">[0x00007f9279235d48] fetch_all() /data/release/www.lishiming.net/<span class="built_in">source</span>/class/table/table_forum_thread.php:523</div><div class="line">[0x00007f9279218f48] fetch_all_search() /data/release/www.lishiming.net/<span class="built_in">source</span>/module/forum/forum_forumdisplay.php:637</div><div class="line">[0x00007f9279218050] +++ dump failed</div><div class="line"></div><div class="line">[24-Oct-2013 00:06:07]  [pool www.lishiming.net] pid 22624</div><div class="line">script_filename = /data/release/www.lishiming.net/forum.php</div><div class="line">[0x00007f9279237938] mysql_query() /data/release/www.lishiming.net/<span class="built_in">source</span>/class/db/db_driver_mysql.php:147</div><div class="line">[0x00007f9279237248] query() /data/release/www.lishiming.net/<span class="built_in">source</span>/class/discuz/discuz_database.php:136</div><div class="line">[0x00007f9279236dc0] query() /data/release/www.lishiming.net/<span class="built_in">source</span>/class/discuz/discuz_database.php:100</div><div class="line">[0x00007f9279235d48] fetch_all() /data/release/www.lishiming.net/<span class="built_in">source</span>/class/table/table_forum_thread.php:523</div><div class="line">[0x00007f9279218f48] fetch_all_search() /data/release/www.lishiming.net/<span class="built_in">source</span>/module/forum/forum_forumdisplay.php:637</div><div class="line">[0x00007f9279218050] +++ dump failed</div><div class="line"></div><div class="line">[24-Oct-2013 00:06:18]  [pool www.lishiming.net] pid 22624</div><div class="line">script_filename = /data/release/www.lishiming.net/forum.php</div><div class="line">[0x00007f9279237938] mysql_query() /data/release/www.lishiming.net/<span class="built_in">source</span>/class/db/db_driver_mysql.php:147</div><div class="line">[0x00007f9279237248] query() /data/release/www.lishiming.net/<span class="built_in">source</span>/class/discuz/discuz_database.php:136</div><div class="line">[0x00007f9279236dc0] query() /data/release/www.lishiming.net/<span class="built_in">source</span>/class/discuz/discuz_database.php:100</div><div class="line">[0x00007f9279235d48] fetch_all() /data/release/www.lishiming.net/<span class="built_in">source</span>/class/table/table_forum_thread.php:523</div><div class="line">[0x00007f9279218f48] fetch_all_search() /data/release/www.lishiming.net/<span class="built_in">source</span>/module/forum/forum_forumdisplay.php:637</div><div class="line">[0x00007f9279218050] +++ dump failed</div></pre></td></tr></table></figure>
<p>脚本：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div></pre></td><td class="code"><pre><div class="line"><span class="meta">#! /bin/bash </span></div><div class="line"></div><div class="line">slow_log=/usr/<span class="built_in">local</span>/php/<span class="built_in">log</span>/php.slow</div><div class="line"></div><div class="line">d=`date -d <span class="string">"-1 minute"</span> +%H:%M`</div><div class="line"></div><div class="line">d_h=`date +%H`</div><div class="line"></div><div class="line">d_m=`date +%M`</div><div class="line"></div><div class="line">d_d=`date +%Y%d`</div><div class="line"></div><div class="line">d_d2=`date -d <span class="string">"-1 day"</span> +%Y%d`</div><div class="line"></div><div class="line">logdir=<span class="string">"/log/php_slow/<span class="variable">$d_d</span>"</span></div><div class="line"></div><div class="line">logdir2=<span class="string">"/log/php_slow/<span class="variable">$d_d2</span>"</span></div><div class="line"></div><div class="line">[ -d <span class="variable">$logdir</span> ] || mkdir -p <span class="variable">$logdir</span></div><div class="line"></div><div class="line">[ -d <span class="variable">$logdir2</span> ] || mkdir -p <span class="variable">$logdir2</span></div><div class="line"></div><div class="line"><span class="keyword">if</span> [ <span class="variable">$d_m</span> -eq <span class="string">"00"</span> ]; <span class="keyword">then</span> </div><div class="line"></div><div class="line">    d1=`date -d <span class="string">"-1 hour"</span> +%H`</div><div class="line"></div><div class="line">    n1=`grep -n <span class="string">" <span class="variable">$d1</span>:[0-9][0-9]:"</span> <span class="variable">$slow_log</span>|head -n1 |awk -F<span class="string">':'</span> <span class="string">'&#123;print $1&#125;'</span>`</div><div class="line"></div><div class="line">    n2=`wc -l <span class="variable">$slow_log</span> |awk <span class="string">'&#123;print $1&#125;'</span>`</div><div class="line"></div><div class="line">    n3=$[<span class="variable">$n2</span>-<span class="variable">$n1</span>]</div><div class="line"></div><div class="line">    tail -n <span class="variable">$n3</span> <span class="variable">$slow_log</span>&gt;/tmp/1.txt</div><div class="line"></div><div class="line">    sed <span class="string">'s/\[0x.*\]//g'</span> /tmp/1.txt |xargs &gt; /tmp/2.txt</div><div class="line"></div><div class="line">    n=`grep <span class="string">'\[pool'</span> /tmp/1.txt|wc -l`</div><div class="line"></div><div class="line">    <span class="keyword">for</span> i <span class="keyword">in</span> `seq 1 <span class="variable">$n</span>`; <span class="keyword">do</span> awk -F <span class="string">'+++ dump failed'</span> <span class="string">'&#123;print $'</span><span class="string">"<span class="variable">$i</span>"</span><span class="string">'&#125;'</span> /tmp/2.txt; <span class="keyword">done</span> &gt; /tmp/3.txt</div><div class="line"></div><div class="line">    <span class="keyword">if</span> [ <span class="variable">$d_h</span> -ne <span class="string">"00"</span> ]; <span class="keyword">then</span></div><div class="line"></div><div class="line">        sed <span class="string">'s/^.*script_filename = //'</span> /tmp/3.txt |grep -v <span class="string">'^$'</span>|sort |uniq -c |sort -rn &gt; <span class="variable">$logdir</span>/<span class="variable">$d1</span>\_slow_log</div><div class="line"></div><div class="line">    <span class="keyword">else</span></div><div class="line"></div><div class="line">        sed <span class="string">'s/^.*script_filename = //'</span> /tmp/3.txt |grep -v <span class="string">'^$'</span>|sort |uniq -c |sort -rn &gt; <span class="variable">$logdir2</span>/<span class="variable">$d1</span>\_slow_log</div><div class="line"></div><div class="line">        sed <span class="string">'s/^.*[0-9] \//\//'</span> <span class="variable">$logdir2</span>/*_log |sort |uniq -c |sort -rn &gt; <span class="variable">$logdir2</span>/<span class="variable">$d_d2</span>\_slow_log</div><div class="line"></div><div class="line">    <span class="keyword">fi</span></div><div class="line"></div><div class="line"><span class="keyword">fi</span></div></pre></td></tr></table></figure>

	

	
		<span class="different-posts"><a href="/2017/10/12/Shell 练习/9. shell 练习-日志汇总/" onclick="window.history.go(-1); return false;">⬅️ Go back </a></span>

	

</article>

	</main>

	<footer class="footer">
	<div class="footer-content">
		
	      <div class="footer__element">
	<p>Hi there, <br />welcome to my Blog glad you found it. Have a look around, will you?</p>
</div>

	    
	      <div class="footer__element">
	<h5>Check out</h5>
	<ul class="footer-links">
		<li class="footer-links__link"><a href="/archives">Archive</a></li>
		
		  <li class="footer-links__link"><a href="/atom.xml">RSS</a></li>
	    
		<li class="footer-links__link"><a href="/about">about page</a></li>
		<li class="footer-links__link"><a href="/tags">Tags</a></li>
		<li class="footer-links__link"><a href="/categories">Categories</a></li>
	</ul>
</div>

	    

		<div class="footer-credit">
			<span>© 2017 失落的乐章 | Powered by <a href="https://hexo.io/">Hexo</a> | Theme <a href="https://github.com/HoverBaum/meilidu-hexo">MeiliDu</a></span>
		</div>

	</div>


</footer>



</body>

</html>
